Appendix A for a detailed description about the architectures.

The identity matrix is missing. This is everywhere in the article / supplementary.

We study the residual bootstrap (RB) method in the context of high-dimensional linear regression.

We study the residual bootstrap (RB) method in the context of high-dimensional linear regression.

We study the residual bootstrap (RB) method in the context of high-dimensional linear regression.

We study the residual bootstrap (RB) method in the context of high-dimensional linear regression. When regression coefficients are estimated via least squares, classical results show that RB consistently approximates the laws of contrasts, provided that $p\ll n$, where the design matrix is of size $n\times p$. Up to now, relatively little work has considered how additional structure in the linear model may extend the validity of RB to the setting where $p/n\asymp 1$. In this setting, we propose a version of RB that resamples residuals obtained from ridge regression. Our main structural assumption on the design matrix is that it is nearly low rank --- in the sense that its singular values decay according to a power-law profile.

In decision making problems for continuous state and action spaces, linear dynamical models are widely employed. Specifically, policies for stochastic linear systems subject to quadratic cost functions capture a large number of applications in reinforcement learning. Selected randomized policies have been studied in the literature recently that address the trade-off between identification and control. However, little is known about policies based on bootstrapping observed states and actions. In this work, we show that bootstrap-based policies achieve a square root scaling of regret with respect to time. We also obtain results on the accuracy of learning the model's dynamics. Corroborative numerical analysis that illustrates the technical results is also provided.

The bootstrap [15] is a ubiquitous tool in applied statistics, allowing for inference when very little is known about the properties of the data-generating distribution. The bootstrap is a powerful tool in applied settings because it does not make the strong assumptions common to classical statistical theory regarding this data-generating distribution. Instead, the bootstrap resamples the observed data to create an estimate, ˆF, of the unknown data-generating distribution, F. ˆF then forms the basis of further inference. Since its introduction, a large amount of research has explored the theoretical properties of the bootstrap, improvements for estimating F under different scenarios, and how to most effectively estimate different quantities from ˆF (see the pioneering [6] for instance and many many more references in the book-length review of [8], as well as [61] for a short summary of the modern point of view on these questions). Other resampling techniques exist of course, such as subsampling, m-out-of-n bootstrap, and jackknifing, and have been studied and much discussed (see [16], [31], [53], [5], and [18] for a practical introduction). An important limitation for the bootstrap is the quality of ˆF. The standard bootstrap estimate of F based on the empirical distribution of the data may be a poor estimate when the data has a nontrivial dependency structure, when the quantity being estimated, such as a quantile, is sensitive to the discreteness of ˆF, or when the functionals of interest are not smooth (see e.g [6] for a classic reference, as well as [3] or [14] in the context of multivariate statistics).

This paper examines the use of a residual bootstrap for bias correction in machine learning regression methods. Accounting for bias is an important obstacle in recent efforts to develop statistical inference for machine learning methods. We demonstrate empirically that the proposed bootstrap bias correction can lead to substantial improvements in both bias and predictive accuracy. In the context of ensembles of trees, we show that this correction can be approximated at only double the cost of training the original ensemble without introducing additional variance. Our method is shown to improve test-set accuracy over random forests by up to 70\% on example problems from the UCI repository.